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Abstract 


Naval  vessels,  like  most  large-capital  projects/programs,  have  a  long  history 
of  cost  growth  and  overruns.  To  get  a  handle  on  this  problem,  NAVSEA’s  Cost 
Engineering  and  Industrial  Division,  NAVSEA  05C,  has  introduced  Probabilistic  Cost 
Risk  Analysis  (PCRA)  into  the  Department  of  Defense  (DoD)  Planning, 
Programmatic,  Budgeting,  and  Execution  System  (PPBES).  The  quantification  of 
cost  in  terms  of  Cumulative  probability  Distribution  Functions  (CDF)  or  “S-curves” 
provides  a  macroscopic  view  of  project/program  risk.  Risk  curves  alone  do  not 
provide  adequate  visibility  into  the  individual  project  risk  drivers;  therefore,  they  are 
insufficient  for  planning  and  managing  Risk  Reduction  Activities  (RRA).  Complex 
projects  typically  involve  a  set  of  high-consequence,  project-specific  risks  that 
require  detailed  analysis  and  for  which  risk  response  actions  need  to  be  developed 
and  implemented.  The  analysis  of  specific  risks  and  RRAs  requires  a  microscopic 
view.  We  present  a  practical  and  mathematically  sound  approach  using  scenarios 
and  Monte  Carlo  simulation  within  the  framework  of  decision  trees  and  risk  curves. 
The  approach  is  detailed  using  a  realistic  but  simplified  case  of  a  project  with  three 
technical  risks. 

Keywords:  Probabilistic  Cost  Risk  Analysis  (PCRA),  Planning, 

Programmatic,  Budgeting,  and  Execution  System  (PPBES),  Cumulative  probability 
Distribution  Functions  (CDF),  Risk  Reduction  Activities  (RRA),  Monte  Carlo 
simulation 
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Introduction 


Cost  growth  has  been  a  major  problem  for  the  US  Navy.  Over  the  past  four 
decades,  the  growth  of  US  Navy  ship  costs  has  exceeded  the  rate  of  inflation.  In  the 
past  50  years,  annual  cost  escalation  rates  for  amphibious  ships,  surface 
combatants,  attack  submarines,  and  nuclear  aircraft  carriers  have  ranged  from  7  to 
11%  (Arena,  Blickstein,  Younossi  &  Grammich,  2006).  Along  with  real  cost  growth, 
the  DoD  has  had  significant  problems  with  cost  estimates.  By  and  large,  the  DoD 
and  the  military  services  have  underestimated  the  acquisition  cost  new  weapon 
systems.  A  recent  study  by  RAND  (Arena,  Leonard,  Murray  &  Younossi,  2006) 
indicates  that  there  is  a  systematic  bias  toward  underestimating  weapon  system 
costs  and  substantial  uncertainty  in  estimating  the  final  cost  of  a  weapons  system. 

The  DoD  recognizes  that  uncertainty  is  an  important  part  of  cost  estimating. 
During  a  2007  seminar  with  a  naval  aviation  program  official,  the  Assistant  Secretary 
of  the  Navy  for  Research,  Development  and  Acquisition,  Dr.  Etter  (Burgess,  2007,  p. 
42),  stated: 

Program  managers  not  only  need  to  know  a  realistic  cost  estimate  for  their 
program,  they  need  to  know  the  percent  probability  of  achieving  that  target. 
For  example,  a  ship  with  a  40%  chance  of  coming  in  on  budget  has  a  60% 
chance  of  being  over  budget.  Such  a  situation  should  prompt  the  project 
manager  to  seek  help  from  the  acquisition  community. 

There  is  an  ongoing  major  shift  in  R&D  and  complex  engineering 
projects/programs  from  deterministic  to  probabilistic  approaches.  (For  convenience, 
in  this  report  we  use  project  to  refer  to  both  project  and  program.)  Probabilistic  Cost 
Risk  Analysis  (PCRA)  provides  the  proper  framework  for  handling  the  many  different 
elements  of  cost  uncertainties,  including  project-specific,  high-consequence  risks. 
These  risk  drivers  must  be  identified,  assessed,  mitigated,  and  controlled  through 
formal  risk  management,  which  is  an  essential  and  critical  discipline,  implemented  in 
today’s  DoD  projects.  The  Risk  Management  Guide  for  DoD  Acquisition  (2006,  p.  3) 
reads: 
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Risk  management  is  a  continuous  process  that  is  accomplished  throughout 
the  life  cycle  of  a  system.  It  is  an  organized  methodology  for  continuously 
identifying  and  measuring  the  unknowns;  developing  mitigation  options; 
selecting,  planning,  and  implementing  appropriate  risk  mitigations;  and 
tracking  the  implementation  to  ensure  successful  risk  reduction.  Effective  risk 
management  depends  on  risk  management  planning;  early  identification  and 
analyses  of  risks;  early  implementation  of  corrective  actions;  continuous 
monitoring  and  reassessment;  and  communication,  documentation,  and 
coordination. 

The  DoD  risk  management  process  is  consistent  with  the  American 
Association  of  Cost  Engineers  (AACE)  approach,  which  includes  identifying  and 
analyzing  risk  factors  or  drivers,  mitigating  the  risk  drivers  when  appropriate, 
estimating  their  impact  on  plans  and  monitoring  and  controlling  risk  during  execution 
(Hollmann,  2006).  Figure  1  depicts  PCRA  as  an  integral  part  of  the  risk 
management  process.  To  be  effective,  PCRA  should  interface  with  each  of  the  risk 
management  activities  and  be  regularly  updated  to  reflect  the  risk  registry,  risk 
mitigation  plans,  and  their  status  integrated  with  the  project’s  Earned  Value 
Management  System  (EVMS)  (NDIA-PMSC,  2005). 
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Figure  1.  Probabilistic  Cost  Risk  Analysis  as  an 
Integral  Part  of  the  Risk  Management  Process 

(Adapted  from  Risk  Management  Guide  for  DoD  Acquisition,  2006) 


The  emphasis  on  risk  management  supports  efforts  to  reduce  lifecycle  costs 
of  system  acquisitions.  An  often-neglected  concept  in  project  risk  management  is 
the  consideration  of  the  entire  project  lifecycle.  Analysis  of  risk  over  the  lifecycle  of  a 
system  can  yield  substantial  benefits.  Conversely,  ignoring  important  stages  of  the 
lifecycle  can  lead  to  substantial  problems  in  terms  of  risk  for  product  development  at 
the  beginning  of  the  lifecycle  and  for  product  upgrade  or  replacement  at  the  end 
(Pennock  &  Haimes,  2001). 


Many  sources  of  cost  uncertainty  in  naval  vessel  construction  such  as 
economic/business  factors  (rates-wages,  overhead,  vendor/supplier  stability, 
inflation  indices,  multi-year  assumptions,  etc.),  learning/rate/curve  assumptions,  and 
cost-reduction  initiatives  are  well  understood  within  the  framework  of  a  macroscopic 
perspective.  These  are  effectively  modeled  with  classical  Probability  Distribution 
Functions  (PDF)  such  as  the  triangular,  Beta,  lognormal,  and  Weibull  distributions. 
These  PDFs,  however,  do  not  model  some  of  the  negative  influences  of  the 
acquisition  process  and  human  behaviors  such  as  the  MAIMS  principle,  “Money 
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Allocated  Is  Money  Spent”  (Kujawski,  Alvaro,  &  Edwards,  2004).  The  MAIMS 
principle  is  the  money-analog  of  Parkinson's  Law.  It  captures  the  fact  that  cost 
under-runs  are  rarely  available  to  protect  against  cost  overruns  while  task  overruns 
are  passed  on  to  the  total  project  cost.  The  PDF  for  each  cost  element  is  modified 
to  set  all  random  values  less  than  its  allocated  value  to  the  allocated  value.  This  has 
important  implications  for  contingency  and  risk  management  (Kujawski,  2007).  A 
realistic  probabilistic  analysis  requires  more  than  PDFs.  There  are  dependencies  or 
correlations  between  cost  elements;  these  need  to  be  modeled  using  correlation 
matrices  (Book,  2001 ).  However,  even  accounting  for  these  additional  effects, 
macroscopic  factors  constitute  only  a  fraction  of  current  typical  project  risk  drivers 
and,  therefore,  cost  uncertainty. 

The  development  and  acquisition  of  naval  vessels,  like  most  complex 
engineering  projects,  is  also  susceptible  to  project-specific  risk  drivers,  such  as  low 
Technology  Readiness  Level  (TRL);  high  design,  manufacturing,  and  complexity; 
significant  requirement  changes;  sizeable  quantity  changes;  large  funding 
uncertainty;  severe  acts  of  nature;  and  serious  accidents.  It  is  tempting  to  assume 
or  claim  that  the  classical  PDFs  that  cost  or  risk  analysts  typically  elicit  for  cost  also 
quantify  the  project-specific,  high-consequence  risks.  Sometimes  these  analysts  go 
through  the  effort  of  identifying  and  discussing  risk  drivers,  but  when  it  comes  to 
quantifying  the  risks  and  estimating  contingency,  they  simply  apply  high/low  ranges 
to  WBS  elements  without  thinking  about  how  a  particular  risk  driver  affects  one  or 
more  cost  elements.  We  think  it  is  invalid  and  counterproductive  to  do  this  because 
it  leads  to  (1 )  the  loss  of  valuable  information  and  visibility  into  important  risks  and 
(2)  a  reduced  focus  and  ability  to  track  risk  mitigation  actions.  Best  practices 
implement  the  identification  of  risk  drivers  and  events  as  the  kick-off  activity  of  cost 
risk  analysis.  Hollmann  (2007)  explicitly  accounts  for  the  risk  drivers  using  their 
Expected  Values  (EV)  in  addition  to  the  macroscopic  classical  PDFs.  He  refers  to 
this  approach  as  the  Driver-Based  Monte-Carlo  (DBM).  This  represents  a  significant 
improvement  over  today’s  typical  Monte  Carlo  analysis.  However,  the  use  of  EV  is  a 
major  limitation  for  use  in  a  truly  probabilistic  framework  in  which  decisions  are 
based  on  a  given  probability  of  success. 
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The  analysis  of  specific  risks  and  risk  response  actions  (RRA)  requires  a 
microscopic  view  and  is  best  carried  out  with  tools  such  decision  trees  (DT), 
influence  diagrams,  or  other  discrete  representations.  This  microscopic  perspective 
offers  many  benefits.  It  is  a  powerful  risk  analysis  method  to  explicitly  model  high- 
consequence  risks  and  RRAs,  thereby  providing  a  tool  for  making  better  decisions. 

It  also  assists  subject-matter  experts  (SMEs)  to  think  about  credible,  high- 
consequence  events  and  better  deal  with  overconfidence  or  optimism  biases. 
However,  the  microscopic  view  is  too  cumbersome  to  individually  analyze  every  risk 
and  source  of  cost  uncertainty.  It  complements  and  needs  to  be  integrated  within 
the  PCRA.  The  total  project  cost  is  then  best  modeled  as  the  sum  of  a  macroscopic 
baseline  cost  and  microscopic  risk  cost  as  depicted  in  Figure  2. 


Figure  2.  Total  Project  Cost  as  the  Sum  of  the 
Macroscopic  Baseline  Costs  and  Microscopic  Risk  Costs 

(Adapted  from  Federal  Transit  Administration,  2004) 

In  this  report,  we  develop  a  practical  and  realistic  integrated 
microscopic/macroscopic  PCRA  method  as  an  integral  entity  of  the  DoD  risk 
management  process,  as  follows: 

1 .  The  cost  and/or  risk  analysts  (simply  referred  to  as  analyst  below)  and 
the  SMEs  jointly  identify  the  individual  risks  using  the  standard  DoD 
risk-identification  process. 
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2.  The  analyst  and  the  SMEs  jointly  screen  the  identified  risks  for  further 
analysis  and  risk  mitigation. 

3.  The  analyst  and  SMEs  jointly  identify  realistic  RRAs  for  the  screened 
risks. 

4.  The  analyst  models  each  risk  and  its  RRAs  using  a  DT. 

5.  The  analyst  works  with  the  SMEs  to  quantify  the  value  of  the  decisions 
and  outcomes  for  each  DT  using  discrete  and  continuous  distributions. 
We  favor  the  Direct  Fractile  Assessment  (DFA)  method  for  data 
elicitation  and  fitting  the  associated  cost  elements  with  a  three- 
parameter  Weibull  distribution. 

6.  The  analyst  quantifies  the  DTs  using  Monte  Carlo  simulation.  Risks 
and  RRAs  are  then  modeled  in  terms  of  risk  curves.  We,  thereby, 
avoid  relying  on  the  minimum  expected  risk  value,  which  is  a  serious 
shortcoming  of  standard  decision  analysis. 

7.  The  analysis  is  readily  performed  using  commercial  Excel  add-ins 
(such  as  Crystal  Ball  ,  @Risk,  etc.)  or  more  specialized  tools  (such  as 
DecisionPro,  Analytica,  etc.) 

8.  The  analysis  is  updated  regularly  and  key  milestones  throughout  the 
project  life-cycle  to  effectively  monitor  and  control  the  performance  and 
selection  of  RRAs  as  old  risks  are  retired  and  new  risks  arise. 

With  the  proposed  approach,  project  managers  and  team  leads  can 
dynamically  determine  the  optimal  temporal  set  of  decision  gates  for  a  given 
probability  of  success,  thereby  reducing  cost  while  increasing  the  probability  of 
project  success. 

Section  2  discusses  the  dynamic  picture  of  project  risks  and  the  need  for  total 
risk  management.  Section  3  presents  the  use  of  Generalized  Decision  Trees  (GDT) 
and  illustrates  their  application  for  the  analysis  of  a  single  risk  with  two  RRAs  as  the 
project  evolves.  Section  4  extends  the  approach  to  multiple  risks  and  illustrates  the 
method  for  the  realistic  but  simplified  case  of  a  project  with  three  technical  risks.  We 
close  the  report  with  some  recommendations  for  further  development. 
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2.  From  Risk  Assessment  to  Total  Risk 
Management 


“Risk  management”  is  an  overloaded  term.  It  is  used  to  denote  both  (1)  the 
activities  following  risk  assessment  and  (2)  the  entire  process  of  risk  assessment 
and  its  management.  Following  Haimes  (1991),  we  use  the  term  “total  risk 
management”  for  the  latter.  The  standard  quantitative  risk  assessment  paradigm 
focused  on  the  following  triplet  of  questions  articulated  by  Kaplan  and  Garrick 
(1981): 


1 .  What  can  go  wrong? 

2.  What  are  the  associated  likelihoods? 

3.  What  are  the  consequences? 

Once  these  critical  questions  have  been  answered,  the  greater  challenge  for 

risk  management  is  to  address  and  control  the  following  three  issues  articulated  by 
Haimes  (1991 ): 


1 .  What  can  be  done  and  what  options  are  available? 

2.  What  are  the  tradeoffs  in  terms  of  costs,  benefits,  and  risks? 

3.  What  are  the  impacts  of  current  decisions  on  future  options? 

Successful  risk  management  requires  that  the  above  three  critical  issues  be 
properly  and  continuously  addressed  to  reflect  the  dynamic  character  of  project  risks 
depicted  in  Figure  3.  The  sources  and  consequences  of  risks  continue  to  evolve 
and  change  over  time.  As  more  information  is  obtained  about  a  particular  risk,  the 
RRA  options  might  change,  thus,  it  is  necessary  to  constantly  monitor  risk.  In 
general,  at  any  point  in  time  there  will  be  a  mix  of  acceptable  and  unacceptable 
results.  The  performance  of  the  RRAs  needs  to  be  monitored  and  controlled  to 
ensure  they  are  adequately  mitigating  risk.  Concurrently,  management  reserves 
need  to  be  reviewed  on  a  periodic  basis  and  dynamically  allocated  where  needed  to 
ensure  project  success  (Kujawski,  2007).  Waldof  (1998)  in  his  risk  management 
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guide  writes,  “Risk  management  efforts  that  fail  do  so  because  the  risk  control 
actions  did  not  keep  up  with  a  changing  program  situation”  (p.  33).  The  proposed 
dynamic  risk  assessment  and  management  approach  provides  a  mathematically 
valid  as  well  as  practical  framework  for  dealing  with  this  challenge. 


Most  Likely 
Cost 
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Figure  3.  The  Dynamic  Picture  of  Project  Cost  Risk 

(Adapted  from  Federal  Transit  Administration,  2004) 
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3.  Generalized  Decision  Trees  for  Modeling  and 
Analyzing  Risk  Response  Actions-Single  Risk  Case 


We  model  and  analyze  each  screened  risk  and  the  associated  candidate 
RRAs  using  a  generalized  GDT.  In  a  GDT,  PDFs  rather  than  discrete  branches  are 
associated  with  the  chance  nodes,  and  the  outcomes  are  analyzed  using  Monte 
Carlo  simulation  (Kujawski,  2002).  This  provides  a  powerful  technique  for  dealing 
with  the  complex  situations  typical  of  today’s  DoD  projects.  It  avoids  bushy  trees 
and  generates  risk  curves,  thereby  removing  the  reliance  of  decision-making  based 
on  expected  value. 

3.1.  Illustrative  Example  #1 — Selecting  the  Initial  RRA 
Selection 

To  illustrate  the  approach,  we  consider  the  single  risk  depicted  in  Figure  4. 

To  be  concrete  and  without  loss  of  generality,  we  associate  Risk  #1  with  the 
fabrication  of  a  complex  module  or  system  and  the  following  two  RRAs:  (1 )  Directly 
fabricate  the  module,  or  (2)  Build  a  prototype  and  then  fabricate  the  module.  The 
GDT  follows  the  standard  DT  representation.  The  decision  nodes  and  chance 
nodes  are  depicted  as  squares  and  circles,  respectively.  The  ordering  of  the 
decision  nodes  corresponds  to  different  temporal  deterministic  events  in  the 
development  and  fabrication  cycle  of  the  module.  The  branches  that  originate  with 
decision  nodes  represent  the  available  RRAs.  The  branches  that  originate  with 
chance  nodes  represent  the  possible  probabilistic  outcomes.  A  descriptive  label,  a 
probability,  and  a  cost  distribution  are  associated  with  each  branch.  These 
probability  and  cost  values  are  conditional  on  the  RRAs  and  may  also  be  conditional 
on  the  outcome  of  other  risks  in  case  of  interdependencies. 
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NOTE:  In  this  hypothetical  case,  the  values  may  be  thought  of  as  $K. 

Figure  4.  Generalized  Decision  Tree  for  Risk  #1 

Figure  4  depicts  the  fabrication  of  a  hypothetical  first  of  a  kind  system.  To 
mitigate  the  associated  risk,  the  project  considers  two  initial  candidate  risk  response 
actions. 

The  data  in  Figure  4  is  as  follows:  The  cost  PDFs  are  three-parameter 
Weibull  PDF  fitted  to  the  10th,  50th,  and  90th  percentiles  determined  in  accordance 
with  the  DFA  method.  The  baseline  cost  is  assumed  to  be  $1,1 00K.  The  cost  risk  is 
then  given  by  the  Value  At  Risk  (VAR)  relative  to  this  value.  For  example, 

VAR*(900,  1,100,  1,400  |  1,100)  denotes  the  cost  risk  distribution  associated  with 
the  three-parameter  Weibull  PDF  with  the  10th,  50th,  and  90th  percentiles  equal  to 
$900K,  $1,1 00K,  and  $1,400K,  respectively,  given  a  baseline  cost  of  $1,1 00K. 

We  evaluate  each  RRA  in  Figure  4  using  the  Excel  Monte  Carlo  simulation 
add-in  Crystal  Ball®.  The  selection  of  a  RRA  is  a  deterministic  event,  and  only  the 
associated  outcomes  can  be  realized.  It  would,  therefore,  be  inappropriate  to  weigh 
or  combine  the  outcomes  of  the  two  RRAs  since  they  are  mutually  exclusive.  The 
PDFs  and  risk  profiles  for  each  individual  RRA  at  the  start  of  the  project  are  depicted 
in  Figures  5a  and  5b,  respectively.  The  PDFs  are  multimodal  and  cannot  be 
represented  using  any  of  the  PDFs  commonly  used  in  risk  analysis  (Vose,  2006). 
The  peak  for  the  “prototype”  RRA  corresponds  to  the  outcome  in  which  the 
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fabrication  of  the  module  fails.  The  PDF  for  the  “direct  fabrication”  RRA  has  two 
modes  corresponding  to  the  sequence  of  events  in  which  the  first  fabrication  and  the 
subsequent  fabrication  following  redesign  both  fail. 

The  Complementary  Cumulative  Distribution  Functions  (CCDF)  or  risk  curves 
shown  in  Figure  5b  provide  a  more  global  picture.  The  exceedance  probability  is  the 
probability  of  exceeding  a  given  consequence  or  (1 — the  probability  of  success).  For 
example,  looking  at  the  VAR(Fab_A2)  curve,  one  reads  that  there  is  approximately  a 
30%  probability  that  the  cost  will  exceed  $1 ,500K.  Equivalently,  one  can  state  that 
there  is  a  70%  probability  that  the  cost  will  be  less  than  $1 ,500K.  The  risk  curve  and 
the  Cumulative  Distribution  Function  (CDF)  carry  identical  information  content. 

Since  we  are  focusing  on  specific  risks  and  VAR,  we  favor  the  risk  curve  or  CCDF 
because,  in  our  opinion,  it  provides  a  better  view  of  the  residual  risk  and 
management  reserve  than  the  S-curve  (or  CDF)  that  typically  represents  the  total 
cost  (including  the  baseline  and  risk  cost  elements). 


Figure  5a.  Probability  Distributions  corresponding  to 
the  Two  RRA  Options  for  Risk  #1 

Given  the  different  scales,  the  two  PDFs  are  shown  separately  for  greater  visibility 
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Risk  #1 ,  RRA  risk  curves 


Figure  5b.  Risk  Curves  for  the  two  RRA  Options  for  Risk  #1 

Please  consider  Figure  5b.  For  any  given  value  on  the  “Value  at  risk”-axis, 
the  risk  curve  that  corresponds  to  the  lowest  exceedance  probability  represents  the 
lower  risk.  Figure  5b  illustrates  that  the  prototype  risk  curve  is  significantly  lower 
than  the  fabrication  risk  curve  and,  thus,  has  less  risk.  In  this  hypothetical  but 
realistic  situation,  the  investment  of  $100K  for  building  a  prototype  provides  a 
significant  return  on  the  investment  as  measured  by  the  significant  risk  reduction. 

To  be  more  precise,  the  prototype  RRA  presents  a  lower  cost  risk  for  all  values 
greater  than  $200K.  For  the  manager  trying  to  decide  if  it  is  worthwhile  to  invest  in 
the  prototype  option,  the  answer  is  to  invest  as  long  as  the  anticipated  benefits  from 
the  prototype  (whether  it  be  cost  savings,  time  savings,  information,  etc.)  exceed 
$200K  and/or  if  the  low-probability/high-consequence  costs  of  direct  fabrication  are 
unacceptable. 

3.2.  Illustrative  Example  #1 — Analyzing  the  Dynamic 
Character  of  Risk 

As  discussed  in  the  previous  section,  we  use  risk  GDTs  to  model  the 
evolution  of  the  potential  RRAs.  Figure  6  depicts  the  Risk  #1  risk  curves  at  the  start 
of  implementation  of  the  “Prototype”  RRA  (dash  line)  and  after  the  successful 
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demonstration  of  the  prototype  (solid  line).  The  latter  risk  curve  moves  to  the  left  of 
the  original  risk  curve  and  is  steeper,  which  reflects  a  reduced  risk.  These  two  risk 
curves  represent  the  value  of  the  unmitigated  risk  exposure  at  two  different  points  in 
time,  thereby  providing  a  metric  for  the  risk  exposure  as  the  project  evolves.  This 
information  is  essential  for  tracking  the  residual  risk  exposure  versus  the  cost  of  the 
expended  RRAs  and  modifying  the  RRAs  as  needed  to  ensure  mission  success.  As 
expected,  the  residual  risk  following  a  successful  prototype  is  less  than  the  original 
risk.  In  contrast,  a  risk  curve  that  moves  to  the  right  of  the  original  risk  curve  means 
that  the  risk  exposure  is  increasing  and  the  selection  of  RRAs  needs  to  be  re¬ 
considered. 


Risk  #1 ,  Prototype  RRA 
Risk  exposure  characteristics 


Figure  6.  Risk  Exposure  Characteristics  for  Risk  #1  with  the  Development  of  a 
Prototype  at  the  Start  of  Risk  Mitigation  and  after  Successful  Demonstration 
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4.  The  Quantification  of  Multiple  Project  Risks 


Consider  a  project  with  n  credible,  high-consequence  risks  {RJ.  Each  risk,  Rj, 
is  characterized  by  a  probability  of  occurrence  pi  and  a  spectrum  of  possible 
outcomes  with  a  PDF  Lj(x),  where  x  is  a  random  variable  that  represents  the 
magnitude  of  the  associated  cost  or  loss.  One  may  then  think  of  this  set  of  risks  as 
a  risk  portfolio  or  repository  (Kujawski  &  Miller,  2007)  with  a  generalized  discrete 
PDF  Rs(x)  given  by: 


RsO)  =  1  (Pi>  Lj(x)),(p2,  L2(x)),...,(pn,  Ln(x)),/l- Jp;,0 


(1) 


i=l 


As  depicted  in  Figure  2,  the  total  project  cost  is  a  random  variable  that 
consists  of  the  sum  of  the  m  base  cost  elements  and  the  explicitly  identified  risk 
costs.  Depending  on  the  state  of  knowledge  of  the  data,  the  base  cost  elements  BCj 
may  be  modeled  as  either  point  estimates  or  continuous  PDFs.  The  total  project 
cost  TC  is  then  the  probabilistic  sum  of  the  m  base  cost  elements  and  n  risk-driver 
costs: 


TCjx )  =  jBCjx)  piLijx )  (2) 

r=i  i= i 

Equations  (1)  and  (2)  provide  visibility  into  the  link  between  the  credible,  high- 
consequence  risks  {RJ  and  the  total  project  cost  risk  curve.  Monte  Carlo  simulation 
tools  such  as  Crystal  Ball®  and  @Risk  can  also  provide  tornado  charts  that 
conveniently  quantify  the  importance  of  the  various  risk  drivers  and  their  link  to  the 
overall  cost  risk.  Projects  can  use  this  information  to  rationally  identity  risks.  This  is 
in  sharp  contrast  with:  (1 )  the  use  of  point  estimates  that  are  at  best  ambiguous 
because  overly  confident  staff  provide  low  cost  estimates,  while  others  may  inflate 
their  cost  estimates  to  make  it  easier  to  achieve  success,  (2)  decision-making  based 
on  qualitative  assessments,  and  (3)  the  consideration  of  only  S-curves,  which  only 
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provide  a  macroscopic  and  somewhat  “black  box”  view  of  project  risk  and  cost 
uncertainty. 

4.1.  Illustrative  Example  #2 — A  Project  with  Multiple  Risks 

Consider  the  hypothetical  project  with  the  following  three  independent  risks: 
Risk  #1  is  depicted  in  Figure  4;  Risks  #2  and  #3  are  depicted  in  Figure  7a  and  7b, 
respectively.  This  example  is  both  rich  and  simple  enough  to  illustrate:  (1)  several 
diverse  RRAs  and  their  analysis,  (2)  the  dynamic  nature  of  the  risk  picture,  and  (3) 
the  monitoring  of  individual  risks  and  allocation  of  management  reserves.  The 
approach  readily  extends  to  dependent  risks  by  incorporating  probability  and 
outcome  values  that  reflect  causality  effects  or  correlations  among  the  risks. 

Risk  #2,  which  is  depicted  in  Figure  7a,  may  be  thought  of  as  a  prime 
contractor  who  subcontracts  the  engineering  and  fabrication  of  a  complex  module 
and  considers  the  following  two  options:  (1 )  subcontract  to  a  single  contractor  A, 
denoted  by  the  branch  PDR_A  associated  with  the  initial  node;  (2)  carrying  two 
subcontractors  and  selecting  the  best  one  for  fabrication  at  the  Preliminary  Design 
Review  (PDR).  We  need  to  uniquely  identify  each  branch.  The  PDR_A  sequence 
represents  the  decision  to  proceed  with  a  single  contractor.  The  PDR_AB  sequence 
represents  the  decision  to  proceed  with  two  contractors  and,  at  PDR,  to  select  the 
best  one  for  manufacturing.  By  selecting  two  contractors  with  different  offerings,  the 
prime  significantly  reduces  the  probability  of  PDR  failure.  The  RW  branches 
represent  the  different  costs  associated  with  rework.  Each  of  these  branches  is 
modeled  with  a  three-parameter  Weibull  distribution  specified  in  terms  of  the  90th, 
50th,  and  10th  percentiles  provided  by  SMEs  or  based  on  relevant  historical  data. 

Risk  #3,  which  is  depicted  Figure  7b,  may  be  thought  of  as  a  prime  contractor 
who  considers  two  different  Verification  and  Validation  (V&V)  strategies  as  a  means 
for  risk  reduction.  The  branch  VVS_1_(Start  or  CDR)  represents  the  use  of  the 
standard  approach  with  planned  expenditures  of  $300K.  The  branch  VVS_2_  (Start 
or  CDR)  represents  the  use  of  a  more  thorough  V&V  strategy  with  planned 
expenditure  of  $1 ,000K.  The  branch  RW_(1  or  2  and  PDR  or  CDR)  represents  the 
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rework  for  V&  V  strategies  1  and  2  following  the  PDR  and  CDR,  respectively.  The 
rework  is  assumed  to  be  inversely  related  to  the  V&V  effort  and  it  is  modeled  with  a 
three-parameter  Weibull  distribution,  as  previously  discussed. 
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Figure  7a.  Generalized  Decision  Tree  for  Risk  #2 
with  Two  initial  Candidate  Risk  Response  Actions 
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NOTE:  The  start  (or  PDR)  and  CDR  periods  are  shown  separately  to  simplify  the  representation  of 
the  sequence  of  events. 

Figure  7b.  Generalized  Decision  Tree  for  Risk  #3 
with  Two  Initial  Candidate  Risk  Response  Actions 

Given  the  above  illustrative  project  with  three  risks  each  and  two  potential 
RRAs,  there  are  eight  possible  initial  Total  Project  RRAs  (TPRRA)  that  the  project 
may  opt  to  implement.  As  previously  discussed,  the  risk  picture  is  dynamic  and  it 
may  give  rise  to  a  large  combinatorial  number  of  outcomes  as  the  project  evolves. 
For  example,  consider  Risk  #1  with  development  of  a  prototype  as  a  risk  reduction 
option.  The  prototype  may  fail  or  succeed,  and  the  fabrication  of  the  final  module 
may  fail  or  succeed.  The  full  representation  of  the  set  of  all  possible  outcomes  for 
even  this  project  is  overwhelming  and  of  limited  value  for  a  research  report.  We 
therefore  limit  ourselves  to  reporting  an  interesting  subset  of  the  complete  analysis, 
as  follows: 


1 .  We  consider  only  two  of  the  eight  TPRRAS. 

a.  Strategy  1.  Use  of  the  lowest  cost-mitigation  option  for  each  risk, 
which  is  equivalent  to  proceeding  as  normal  (i.e.,  do  not 
implement  specific  RRAs  for  any  of  the  three  risks).  This  is  the 
approach  that  a  risk-seeker  project  manager  would  favor. 
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b.  Strategy  2.  Use  the  most  effective  RRA  for  each  risk.  This 
corresponds  to:  (1 )  developing  a  prototype  for  Risk  #1 ;  (2) 
proceeding  with  two  contractors  for  Risk  #2;  and  (3) 
implementing  the  more  thorough  V&V  effort  for  Risk  #3.  This  is 
the  approach  that  a  risk-averse  project  manager  would  favor. 

2.  For  each  strategy,  we  assume  the  best  possible  outcomes  for  the  first 
probabilistic  nodes,  which  for  convenience  we  identify  by  the  time  of 
occurrence  T1 : 

a.  The  Risk  #1  prototype  and  the  Risk  #2  review  PDR_AB  succeed. 

b.  Risk  #3  has  no  gates.  The  risk  reduction  is  directly  accounted  in 

the  magnitude  of  the  rework. 

3.  For  each  strategy,  we  assume  the  worst  outcomes  for  the  probabilistic 
nodes  for  the  first  probabilistic  nodes,  which  for  convenience  we 
identify  the  time  of  occurrence  T 1 : 

a.  The  Risk  #1  prototype  and  the  Risk  #2  review  PDR_AB  fail. 

b.  Risk  #3  has  no  gates.  The  risk  reduction  is  directly  accounted  in 

the  magnitude  of  the  rework. 

Figures  8  and  9  depict  the  initial  and  residual  risks  under  strategies  #1  and  #2 
assuming  the  best  and  worse  T1  outcomes,  respectively.  These  data  provide 
bounds  for  the  risk  range  that  may  threaten  the  project  following  implementation  of 
the  initial  set  of  RRAs.  Figure  10  depicts  this  useful  information,  thereby  avoiding 
the  need  to  analyze  the  full  combinatorial  set  of  RRAs.  The  dynamic  picture  of  risk 
and  the  value  of  monitoring  risk  are  further  discussed  in  the  next  section. 
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Figure  8.  Risk  Exposure  Characteristics  for  Risk-seeking  Strategy  #1  and  a 
Risk-averse  Strategy  #2  Assuming  the  Best  Possible  Outcomes  at  the  2nd 
Decision  Points;  i.e.,  Good  Luck  Prevails  on  the  Project  for  the  Initial  Set  of 

Risk  Response  Actions 
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Figure  9.  Risk  Exposure  Characteristics  for  Risk-seeking  Strategy  #1  and  a 
Risk-averse  Strategy  #2  Assuming  that  Assuming  the  Worse  Possible 
Outcomes  at  the  2nd  Decision  Points;  i.e.  Murphy’s  Law  Prevails  on  the  Project 
for  the  Initial  Set  of  Risk  Response  Actions 


Figure  10.  Bounds  on  the  Project  Risk  Following  Implementation  Risk-seeking 
Strategy  #1  and  a  Risk-averse  Strategy  #2 
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5.  The  Value  of  Analyzing  Risk  Dynamism 


Figures  8  and  9  illustrate  the  value  of  monitoring  the  performance  of  the 
RRAs  and  other  risks.  When  the  best  outcomes  are  realized,  both  strategies  reduce 
risk.  As  depicted  in  Figure  8,  both  T1  curves  move  to  the  left  and  become  narrower 
than  the  start  risk  curves.  Likewise,  if  the  worst  outcome  prevails  as  assumed  in 
Figure  9,  then  both  strategies  actually  increase  the  cost  risk  exposure  of  the  project. 
Thus,  graphing  risk  curves  over  time  provides  a  metric  to  measure  the  success  of 
risk  mitigation  efforts. 

Figures  8,  9,  and  10  provide  additional  information  that  can  help  the  project 
rationally  choose  the  preferred  o  strategy.  Under  the  best-case  scenario  (Figure  8), 
at  the  start  of  the  project,  Strategy  #1  offers  a  lower  risk  exposure  below  $1 ,500K, 
while  Strategy  #2  offers  a  lower  risk  exposure  above  that  value.  Both  strategies  are 
equal  in  terms  of  exceedence  probability  (60  %)  at  the  “breakeven”  point  of  $1 ,500K. 
What  does  the  project  gain  by  extending  the  analysis  to  time  T1  ?  It  gains  the 
information  that  the  “breakeven”  point  is  lower  ($1 ,200K)  and  the  risk  at  that  point  is 
also  lower  (40%).  So,  which  one  is  the  best  choice?  The  optimistic  project  manager 
most  likely  assumes  that  the  best  outcome  would  be  realized,  thereby  making  a 
choice  based  on  the  expected  benefits.  As  long  as  the  expected  benefits  of  the 
RRA  are  greater  than  $1 ,200,  he/she  would  choose  Strategy  #2.  But  of  course, 
there  is  no  such  assurance,  so  we  must  examine  the  worst-case  scenario. 

Figure  9  shows  the  results  of  implementing  each  strategy  over  time  assuming 
the  worst  outcome  (Murphy’s  Law).  As  expected,  the  risk-seeking  Strategy  #1 
significantly  increases  the  project  cost  risk  exposure  when  things  go  bad,  but  the 
more  conservative  Strategy  2  is  much  less  sensitive  to  bad  outcomes.  In  fact,  at  T1 , 
Strategy  #  2  dominates  Strategy  1  (i.e.,  it  has  a  lower  risk  for  any  value.)  For 
pessimists,  the  choice  is  simple:  Strategy  #2  is  especially  effective  in  providing 
insurance  against  the  worst  outcomes. 
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Which  strategy  is  chosen  depends  on  the  decision-maker’s  risk  aversion.  Is 
he/she  an  optimist  or  a  pessimist?  In  either  case,  if  the  expected  benefits  of  risk 
mitigation  exceed  $1 ,200K,  Strategy  #2  is  the  best  choice.  We  believe  examining 
risk  information  in  this  way  provides  useful  insight  and  helps  project  managers  make 
better  choices. 
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6 


Conclusions 


6.1  Lessons  Learned 

This  report  presents  a  method  for  evaluating  and  tracking  project-specific 
risks  at  the  microscopic  level.  This  type  of  analysis,  as  opposed  to  the  macroscopic- 
level  risk  analysis,  is  essential  for  risk  management.  While  the  macro  level  provides 
some  information  about  total  cost  risk,  the  micro  level  allows  the  project  manager  to 
plan  and  control  risk  response  actions  that  influence  total  cost  risk. 

This  report  also  demonstrates  the  use  of  GDTs  to  model  the  evolution  of  the 
potential  RRAs  and  risk  curves  to  evaluate  the  risk.  We  believe  risk  curves  are 
better  than  the  expected-value  results  usually  given  by  traditional  decision  tree 
analysis  because  they  contain  all  the  risk  information  both  in  terms  of  probabilities 
and  value  at  risk.  Risk  curves  derived  from  Monte  Carlo  simulation  on  GDTs  are 
particularly  useful  when  analysts  are  comparing  different  risk-mitigation  strategies. 
The  “breakeven”  points  help  the  risk  manager  understand  the  conditions  under 
which  each  strategy  is  most  appropriate.  Combined  with  scenario  analysis,  it  offers 
an  opportunity  to  make  cost-benefit  tradeoffs  among  strategies.  This  thorough 
approach  allows  management  to  consider  what  they  mean  by  “acceptable”  risk  and 
explicitly  models  the  tradeoff  between  risk  and  benefits  of  any  given  RRA.  Tracking 
the  performance  of  RRAs  over  time  is  key  to  understanding  the  dynamic  nature  of 
risk  management  and  can  reveal  necessary  changes  in  strategy. 

6.2  Future  Research  and  Implementation 

The  cost  risk  analysis  and  management  developed  in  this  report  is  both  rich 
and  simple  enough  to  illustrate:  (1 )  the  portfolio  aspect  of  several  diverse  RRAs  and 
their  analysis,  (2)  the  dynamic  nature  of  the  risk  picture,  and  (3)  the  monitoring  of 
individual  risks  and  allocation  of  management  reserves.  The  approach  readily 
extends  to  dependent  risks  using  different  probability  and  outcome  values  that 
reflect  causality  effects  among  the  risks.  The  authors  think  that  it  provides  the 
detailed  information  that  program  managers  need  and  want  when  they  face  hard 
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decisions  on  programs.  There  is  a  cost  for  this  type  of  analysis,  but  it  is  small 
considering  the  potential  benefits.  The  proposed  approach  is  both  practical  and 
mathematically  valid  and  can  be  implemented  using  commercially  available  tools 
such  as  Crystal  Ball®  and  @Risk. 

The  next  phase  is  to  further  develop  the  methodology  and  an  integrated  tool 
so  that  a  program  manager  can  select  a  portfolio  of  risk  reduction  activities  that 
maximizes  the  cost-benefit  of  such  activities.  The  dynamic  nature  of  the  approach 
will  allow  the  program  manager  to  monitor  and  manage  the  performance  of  risk 
reduction  activities  over  time  and  optimize  the  allocation  of  risk  management 
resources  over  the  life-cycle  of  the  project/program.  The  challenge  is  to  start 
implementing  these  more  refined  cost  models  and  risk  management  practices. 
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